Confidence Intervals in Analysis and Reporting of Clinical Trials

Regulatory agencies around the world have recommended reporting confidence intervals for treatment differences along with the results of significance tests. SAS provides easy and convenient ways to produce confidence intervals using procedures such as PROC GLM and PROC UNIVARIATE in conjunction with ODS (output delivery system). In this paper, I will discuss the relationship between significance tests and confidence intervals, summarize the types of confidence intervals used in clinical study reports, and provide examples from clinical trials to illustrate the computation of distributiondependent confidence intervals for the mean treatment difference and distribution-free confidence intervals for the median response within each treatment group using SAS. INTRODUCTION Confidence interval estimation and significance testing (hypothesis testing) are the two most commonly used statistical inference methods for clinical trials (Walker, 1997). Because p-values have been widely presented and, in the past, more easily obtained from standard statistical software than have confidence intervals, significance tests have also been more widely accepted by the medical community than have confidence intervals. The extensive use of significance tests with clinical trial data has further increased their popularity and made confidence intervals less popular. However, the overuse and misinterpretation of significance tests have lead to advocates for more frequent use of confidence intervals (Simon, 1993). There is a close relationship between confidence intervals and significance tests (Hahn and Meeker, 1991); in fact, a confidence interval can often be used to test a hypothesis. If the 100(1-α)% confidence interval for the mean treatment difference in a clinical trial does not contain zero, there is evidence to indicate a treatment difference at the 100 α % significance level. This strategy is equivalent to the hypothesis test that rejects the null hypothesis of no mean treatment difference at the level of α. Compared to p-values, confidence intervals are generally more informative. They provide quantitative bounds that express the uncertainty inherent in estimation, instead of merely an accept or reject statement. The length of a confidence interval depends on the sample size; this influence of sample size is evident from observing the length of the interval, while this is not the case for a significance test. So confidence intervals are usually more meaningful than statistical hypothesis tests alone. Moreover, they are easier to explain to those with no formal training in statistics (Hahn and Meeker, 1991). Regulatory agencies around the world have emphasized in recent years the importance of reporting confidence intervals in clinical study reports. The ICH Harmonized Tripartite Guideline on statistical principles for clinical trials states, “Estimates of treatment effects should be accompanied by confidence intervals, whenever possible, and the way in which these will be calculated should be identified.... it is important to bear in mind the need to provide statistical estimates of the size of treatment effects together with confidence intervals (in addition to significance tests).” In this paper, I will discuss the types of confidence intervals and the different methods for constructing them. I will also illustrate the calculation of confidence intervals for clinical trial data by providing sample SAS code and output. Finally, I will discuss the advantages of presenting confidence intervals along with p-values in clinical study reports. CHARACTERISTICS OF CONFIDENCE INTERVALS Because there exist a variety of confidence intervals, the analyst must determine which type of interval to use depending on the application (Hahn and Meeker, 1991). Two commonly used types are confidence intervals for population parameters and confidence intervals for distribution percentiles. The most frequently used type of confidence interval attempts to capture the population mean. Sometimes, however, the analyst may construct confidence intervals for the standard deviation or other shape parameters for a distribution to satisfy his needs. In the case that the assumed distribution parameters are not suitable to describe the sampled population, the analyst may focus on one or more percentiles of the sampled distribution and construct confidence intervals for them (for example, for the median or the quartiles). Sometimes onesided confidence bounds are desired in situations where the major interest is restricted to the lower limit or the upper limit alone. All statistical intervals have an associated confidence level. The analyst must determine the confidence level based on what seems to be an acceptable degree of assurance for the specific application (Hahn and Meeker, 1991). For instance, the analyst may construct a 95% confidence interval for the mean. This indicates that the method of construction guarantees that 95% of all such intervals will contain the (true) population mean. (Of course, this also means that 5% of them will not.) One can request a higher level of confidence, which will reduce the chances of obtaining an interval that does not contain the population mean. However, increasing the confidence level results in a wider (that is, less precise) interval for a fixed sample size. On the other hand, when there is a fixed confidence level, the length of intervals becomes shorter as the sample size increases. So, the analyst may choose higher confidence levels with large samples and lower confidence levels with small samples. In some cases, obtaining meaningful confidence intervals becomes impractical because of the small sample size or complexity of analysis. CONFIDENCE INTERVALS FOR CLINICAL TRIALS The most frequently used confidence interval for clinical trial data is the 95% confidence interval for the mean treatment difference. The selection of 95% for the confidence level is common across disciplines. The reason for this selection seems quite obvious for clinical trials, especially for confirmatory trials as a result of the study design, and the 95% confidence level provides reasonable assurance along with adequate precision for most trials. The methods for calculating these confidence intervals can be generally put into two categories: distributiondependent and distribution-free methods. Construction of distribution-dependent confidence intervals requires one to assume a particular distribution, such as the normal distribution. From experience, the normal assumption appears to be valid for many clinical trial data analyses. However, it may be inappropriate to calculate distribution-dependent confidence intervals when the assumed distribution does not fit the data well. In such cases, distribution-free confidence intervals should be constructed. A distribution-free interval sometimes may not exist, and its length is generally longer than the corresponding distribution-dependent interval for a particular distribution. This is the price that one pays for not making the distribution assumption (Hahn and Meeker, 1991). So, a distributiondependent confidence interval should be chosen whenever there is solid evidence that the data follows a tractable distribution. DISTRIBUTION-DEPENDENT CONFIDENCE INTERVAL If the assumption that the data are normally distributed is valid, one can construct confidence intervals for the mean treatment difference. The general form of a confidence interval for the mean difference between two treatment groups (Group A and Group B) is ) ( * , / a Yb Ya df b a S t Y Y − − ± − 2 1 (1) whereY is the mean or least squares mean, and ) ( Yb Ya S − is the standard error of the estimate of ( b a Y Y − ), and df t , / a 2 1 − is the 100(1-α/2) percentile from student’s t distribution with ‘df ‘degrees of freedom. In practice, one can calculate a 95% confidence interval for the mean difference through analysis of variance, although it is critical to obtain the degrees of freedom and estimate the standard error (or mean square error) correctly. Before the release of SAS version 8, the analyst had to extract these values from the SAS procedures and then calculate the confidence intervals using the formula above (or one of its variations) with custom written SAS code. The following code using SAS version 6.09 represents one way to obtain the 95% confidence interval for the mean treatment difference from an ANOVA model. *-------------------------------------*; * The following statements get output *; * datasets containing statistics *; * needed for calculation of confidence*; * interval *; *-------------------------------------*; proc glm data=final outstat=glmdt noprint; class &trt &str &invcd; model &dep=&indp; lsmeans &trt/pdiff stderr tdiff